The Information Bottleneck EM Algorithm
نویسندگان
چکیده
Learning with hidden variables is a central challenge in probabilistic graphical models that has important implications for many real-life problems. The classical approach is using the Expectation Maximization (EM) algorithm. This algorithm, however, can get trapped in local maxima. In this paper we explore a new approach that is based on the Information Bottleneck principle. In this approach, we view the learning problem as a tradeoff between two information theoretic objectives. The first is to make the hidden variables uninformative about the identity of specific instances. The second is to make the hidden variables informative about the observed attributes. By exploring different tradeoffs between these two objectives, we can gradually converge on a high-scoring solution. As we show, the resulting, Information Bottleneck Expectation Maximization (IB-EM) algorithm, manages to find solutions that are superior to standard EM methods.
منابع مشابه
An Analysis of Model-based Clustering, Competitive Learning, and Information Bottleneck
This paper provides a general formulation of probabilistic model-based clustering with deterministic annealing (DA), which leads to a unifying analysis of k-means, EM clustering, soft competitive learning algorithms (e.g., self-organizing map), and information bottleneck. The analysis points out an interesting yet not well-recognized connection between the k-means and EM clustering—they are jus...
متن کاملVRED: An improvement over RED algorithm by using queue length growth velocity
Active Queue Management (AQM) plays an important role in the Internet congestion control. It tries to enhance congestion control, and to achieve tradeoff between bottleneck utilization and delay. Random Early Detection (RED) is the most popular active queue management algorithm that has been implemented in the in Internet routers and is trying to supply low delay and low packet loss. RED al...
متن کاملVRED: An improvement over RED algorithm by using queue length growth velocity
Active Queue Management (AQM) plays an important role in the Internet congestion control. It tries to enhance congestion control, and to achieve tradeoff between bottleneck utilization and delay. Random Early Detection (RED) is the most popular active queue management algorithm that has been implemented in the in Internet routers and is trying to supply low delay and low packet loss. RED al...
متن کاملLearning Hidden Variable Networks: The Information Bottleneck Approach
A central challenge in learning probabilistic graphical models is dealing with domains that involve hidden variables. The common approach for learning model parameters in such domains is the expectation maximization (EM) algorithm. This algorithm, however, can easily get trapped in suboptimal local maxima. Learning the model structure is even more challenging. The structural EM algorithm can ad...
متن کاملApplying the Information Bottleneck Approach to SRL: Learning LPAD Parameters
In this paper, we propose to apply the Information Bottleneck (IB) approach to a sub-class of Statistical Relational Learning (SRL) languages. Learning parameters in SRL dealing with domains that involve hidden variables requires the use of techniques for learning from incomplete data such as the expectation maximization (EM) algorithm. Recently, IB was shown to overcome well known problems of ...
متن کامل